Skip to content

feat(alerts): add opt-in spend, pool and provider health alerts - #46

Merged
ALikesToCode merged 3 commits into
feat/session-tiersfrom
feat/gateway-alerts
Oct 9, 2026
Merged

ALikesToCode merged 3 commits into
feat/session-tiersfrom
feat/gateway-alerts

Conversation

@ALikesToCode

Copy link
Copy Markdown
Owner

Summary

Adds opt-in spend, pool and provider health alerts behind GATEWAY_ALERTS_ENABLED, which defaults to false. With the flag off, /admin/alerts returns 404, nothing is observed or stored, and existing routes, responses and storage are unchanged.

  • Rule kinds:
    • spend: an operator USD budget per UTC day or month, with default thresholds of 85% and 100%.
    • unknown_price: the share of recorded requests that have no price.
    • provider_circuit: an open circuit.
    • pool_exhaustion: no eligible credential is left in the pool.
    • provider_health: consecutive failures within one hour.
  • Content-free evaluation:
    • Rules are evaluated over aggregates only.
    • Payloads label their basis: gateway_cost_estimate, unknown_price_coverage or observed_health. A spend alert is an estimate from recorded usage and configured prices, not a provider invoice.
    • No prompt, response, credential, username or recipient appears in a payload, status response or diagnostic.
    • Alerts never start a generation or a paid health probe. Provider health is read passively from existing circuits, route health and pool eligibility.
  • Admin route:
    • GET/POST /admin/alerts requires an administrator dashboard session and keeps the existing CSRF protection.
    • Updates use the current revision; a conflict returns 409.
    • At most 20 rules; configuration is bounded to 8 KiB.
    • Responses are Cache-Control: no-store.
  • Delivery:
    • GATEWAY_ALERT_WEBHOOK_ALLOWLIST takes up to 20 exact public HTTPS origins on port 443. IP literals, local hostnames, credentials and redirects are rejected.
    • Webhook addresses are stored as private configuration and are never shown in status.
    • The Worker's scheduled run delivers at most ten events per run, with three attempts per event and a three-second timeout per attempt.
    • Events are deduplicated per rule, threshold, period and revision for 15 minutes.
    • History is kept for seven days and capped at 2,000 records.
  • Storage: the new D1 tables gateway_alert_rules and gateway_alert_events (migration 0022). With the flag on and the tables absent, the feature fails closed with 503.

Details are in docs/gateway-alerts.md.

Glue left for the Wave 4 integration PR

  • Schedule runAlertDelivery from the Worker's scheduled handler, with the fixed aggregate collector and the redirect-rejecting transport.
  • Dispatch the private alert state (handleAlertState) behind internal authentication, and add the alerts private endpoint mapping in services/intelligence_d1_store.py.
  • README line.

The Flask registrar already mounts the admin route and exposes gateway_alert_observer, and the free health checks call the observer.

Stacking

This PR is based on feat/session-tiers (#45), so migrations 0018–0022 stay contiguous. Review only the top commit, feat(alerts): add opt-in spend, pool and provider health alerts; the merge commit brings in the lower layers.

Deployment

Apply the migration before merging to main. /ready checks the tables, and a push to main does not apply D1 migrations:

npx wrangler d1 migrations apply multillm-intelligence --remote

Shared registrations

  • Added tests/test_gateway_alerts_worker.mjs to test:worker.
  • Added GATEWAY_ALERTS_ENABLED and GATEWAY_ALERT_WEBHOOK_ALLOWLIST to the Container env keys.
  • Added both alert tables to the D1 readiness tables.
  • Added worker/alert-delivery.mjs to the secret firewall module inventory, and /admin/alerts to the Flask route inventory. The route only talks to the private state endpoint and dispatches to no provider.

Tests

  • New tests: tests/test_gateway_alerts.py and tests/test_gateway_alerts_worker.mjs. They cover:
    • the flag off;
    • malformed metadata;
    • queue capacity;
    • oversized configuration;
    • revision conflicts;
    • Worker status validation.
  • Full Python suite: 3,684 passed, 2 skipped. ruff, mypy and bandit are clean.
  • npm run test:worker: 1,434 passed. npm run test:integration: 7 passed.
  • cf:migrations:check reports 22 contiguous, additive migrations.
  • test:deploy, free routes, client headers and the wrangler deploy --dry-run bundle all pass.
  • Live webhook delivery has not been exercised. The tests use deterministic transports.

CI notes

  • GitHub python/node jobs: the dependency-audit steps fail on every branch, including main. fix(deps): clear the CI dependency audit advisories #42 fixes that.
  • Workers Builds: fails on this stack because the Wave 3 base adds the AdmissionCoordinator Durable Object migration, which wrangler versions upload cannot apply. This is expected until wrangler deploy runs on main.

This is part of Wave 4.

Evaluate operator alert rules over content-free aggregates (spend estimate, unknown-price coverage, open circuits, exhausted pools, consecutive provider failures) behind GATEWAY_ALERTS_ENABLED (default off). Rules and delivery events live in D1 (migration 0022). The Worker's scheduled run delivers deduplicated webhooks to allowlisted HTTPS origins (GATEWAY_ALERT_WEBHOOK_ALLOWLIST), at most three attempts per event. Administrators manage rules at /admin/alerts, which returns 404 while the flag is off.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8ce72ab8-99f2-4d18-8856-3d1f3f32990b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gitguardian

gitguardian Bot commented Oct 9, 2026

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
37552414 Triggered Generic Password f1e4188 worker/alert-delivery.mjs View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
❌ Deployment failed
View logs
multillm-proxy 6487bac Oct 09 2026, 12:42 AM

@ALikesToCode
ALikesToCode merged commit 502dcc8 into feat/session-tiers Oct 9, 2026
3 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant